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(54) Titte: MOTION VECTOR ESTIMATION AND DETECTION OF COVERED/UNCOVERED IMAGE PARTS 

(57) Abstract 

In a method of detecting 
covered and uncovered parts in an 
image to be interpolated between 
neighboring previous (I) and 
next (ill) input images, backward c 
motion vectors from the next ' 
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input image to the previous input 
image, and having corresponding 
backward estimation errors, and 
forward motion vectors from 
the previous input image to the 
next input image, and having 
corresponding forward estimation 
errors, are determined, uncovered 
and covered parts are detected 
in the neighboring previous and | 
next input images, respectively, in 
the thus detected uncovered parts, 
uncovered parts in the image to 
be interpolated are detected by 

determining second backward estimation errors by comparing both neighboring previous and next input images when partially shifted over 
the backward motion vectors to a temporal location of said image to be interpolated, and by comparing the second backward estimation 
errors to a threshold, and in the thus detected covered parts, covered parts in the image to be interpolated are detected by determining 
second forward estimation errors by comparing both neighboring previous and next input images when partially shifted over the forward 
motion vectors to a temporal location of the image to be interpolated, and by comparing the second forward estimation errors to the 
threshold. 
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Motion vector estimation and detection of covered/uncovered image parts. 



The invention relates to motion vector estimation and detection of 
covered/uncovered image parts. The invention also relates to motion-compensated 
interpolation, and to a display apparams comprising means for carrying out such a motion- 
CQl compensated interpolation. 



i 



The paper "Motion estimation method using the spatio-temporal 
characteristics of moving objects", presented at the International Workshop on HDTV, Los 
Angeles, October 1996, by H. Sonehara et al., discloses a motion vector detection method 

10 which takes account of the spatio-temporal continuity of motion vectors and the local 

characteristics of pictures. Motion vector trajectories are traced to assess the spatio-temporal 
continuity of motion vectors. To remove erroneous motion vectors detected in the 
covered/uncovered background of the contour area of moving objects, motion vectors are 
estimated by means of a detection-area hierarchy and the correlation of luminance signals. It 

15 is necessary to identify, in two sequential fields, the area (covered and uncovered background 
area) in which the picture correlation becomes indeterminate. For this, a detection block is ; 
set in the current field and vectors are detected in relation to the next field. Then a detection 
block is set in the next field and vectors are detected in reliation to \h€ currehf * field. In short, 
niotion vectors are detected two-way. 

20 ■ ^ ' EP-B-0,535,066 discloses a method of assigning the appropriiate one (if 

any) of a plurality of possible motion vectors to pixels of a video signal consisting of a series 
of pictures, the method. comprising the steps of: a backwards .assignment operation -i^ which! 
a first input picmre is shifted by each of a plurality of possible vectors and coihpaired to a ; 
' Second, succeeding picmre to determine assignment errors for each vector at each< pixel; ra 

25 / 'forward assignment operation:in which the second picture is shifted by each' of a; plurality of 
possible Vectors and coniparedvp tfe first picture jlo 'dete^^ errbrjs^fbr each ; 

vector at each pixel; and using for each pixel the assignment errors determined in jboth \ • 
forward assigiunem and biackwards assignment operations to determine whether any of the 
vectors is appropriate for that pixel. .The method may further comprise comparing the - ^ 
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assignment errors for i^^ assignment operations for a pixel to 

determine whether the pixel may relate to revealed or obscured background/ This combined 
vectdr assignment flags regions of both i-eveiled"and obiscured* background as 'no match' so 

' that: a fallback ihteipblatiori techniq^^ usied. This ifallback" interpolation technique has 

. . ■ I. . • 

5 not bedri described. Iri an alternative methd from the^ instant of an input picture are 

craverted to an ouipiit j^icture at^ a differe^^ in time by projecting: the input vectors 

foiwai'd in ti'm^ their m6ti6h)i'^Severarpr6bl^^ 

vectoW fdii^ intiriie: While these problems' aifS 
foi^aril aiid'^b^ckwi^^ is siriipler^jahd-th^ 

............ . . ^ o 

' ' ' It is, mr£'r a//a, an objecrof the^ i^^ m 

detection / motion-compensated interpolatidn. To this end, a first aspect Of the invention m 
provides a niotiori vector eistimation methbld' aiid device -as defined in claims 1 and 5. A j 
15 second aspect of the invention provides^ an ihtefpidlation method as defined in-claim 4. A ^ 
third aspect of the invention provides a display apparams as defined in claim 6. A fourth ^ 
aspect of the invention provides a method of and device for detecting covered and uncovered ^ 
parts in an image to be interpolated as defined in blSiin 7 arid 9. A fifth aspect of the 
invention provides a mbtiori Claim 10. Advantageous 

20 embodiiTierits Srfie defined^'^i^ - 

A primary aspect of the invention provides a method of detecting covered 
arid uncbveredf jiarts in ah iriiage to be interpolated '^between neighboring previous and next 
input images, in which backward motion vectors from the next input image to the previous 
input image, and having corresponding backward estimation errors, and forward motion 
' 25 vectors from the' prfeviotis input image to the next input image; and having corresponding 
forward estimatioii errors, are determined, uncovered' and covered parts are detected in the 
lieighiDormg previous an next input images, respectively , in -the thu detected' uncovered 
parts, uncovered parts in the irriage to be interpolated are detected by determining second 
* backward ehimation errors by comparing both neighboring previous and next . input images 
30 'wlien partially shifte^^ the backward motion vectors to a temporal location of said 'image 
' to be'interpoiated, ^^nd by comparing tfie sec6y--ba6kWard estimation 
" and in the thus detected covered' pahs, 'covered parts in" the are 
detected by" determining s'^corid fdh^ estimation' errors fey ^domparin^^ 
previous and riext input images when partiaHy shifted over the forward motion' Vectors to a 
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.temporal location: of the image to.be inteipolated,.and b s^cpncj fovvji^rd 

* -esiiiriaiion errors to the threshold. ^ • j m . ; ^ . ^ 

: ; r 'Anoiher .aspect of .the jnyem^^ a method of estimating motion 

. . vectors: for an: imzige to be interpolated betv^een , neighboring previous aiid next jnpm images, 
5 . in which.moiion from the. next i^put iHiage-io the previous input ifP^ge i?^.dQtea^^ obtain 
backward .motionrvecjt^^^^ haying; GqrFpsponding backw^^ errojs, mpq^ the 

' previpus£ inputs image^to .the :next anput image , is detected to obtain for^y^r^^^ 
..:having:.cofTesp.onding fprvifard^ and;Covey:^d Rarts ar?^^^^^^^ in 

the image to ..be 4nten>Ql?ied^ tmotio^ 
10 dependence upon the forward motion vectors taken from locations wholly and/or partly. 

shifted over the backward motion vector, and motion vectors for the covered parts are 
• ; furnished in dependence: upon the^Jbackward^mpti vectors taken from locations wholly 
i,./. and/or fpanly /Shifted ;pver the. fprw^^^ vectors. , , 

r , , : , These and other aspects, of the iny^^^^ 
15 ' elucidated, with reference, to the embodinients ^escribed ^^h^^ , -. ^ . . :^ 



V... :v f . In the drawings; ■iv,-^...^^. r ... ; • : > > . ... . 

: u > - J. » ^t... - Fig.. 1 illustrates covering and. un situations; 
20 Fig. 2 illustrates, where, in the interpolated image covering and uncovering 



; occurs and 



: Fig. .3- shows a prefeirxeid embodiment of the purrent invention. 



25 \ . 7 . : . . - A primary aspect of .the present invention aims at a strpng reduction, if 

' -not: an elimination, of halo effecte for, systems. fhatt have acces? tp both neighl^orin^ fields by 
' . analyzing . in which, of the two fields the.: in^^ present 
"r: : . • (Le;: not-covered) using two m[otion^vectoi*s per.,locatipn. This infprmation is used to 
r^.: t interpolate thCvmotion cpmp^^^^^ picture frprn .both .neighbori^^ pixels 

:30. from either of the two images that wiere, either coyer^^^ 
. . o.^vi ^jestimatprsiin ^parallel is cxmsi^ ?y^^ ^Pi^?^^^^[,P^^^^^'<f^^^ earlier 

s .^: worfo[7;:8] Aas T^duced an^ affordable level. Nevertheless, 

: ' ■ ^l; m•alternali^^^^ the. additional .calculations are only required in — 

: . , / t. whetre. covering .or uncovering is -expected, i.e. near picture parts; Jii which discontinuities 



areas 
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(contours) in the:vecior field occur; j '^.it^^ ^ . ..^ : 

Basic requirement for the current algorithm is the availability of t\yo 

motion vectors for every location. The first vector J5, is the one calculated by shifting a 
previous field to optimally match a current field, whereas the second is obtained by. shifting 

5 the data from the current field to optimally match a previous field. The required 

segmentation of the image in "normal, "covering", and "uncovering" regions will be dealt 
with later! Fir^t, we shall elucidate the nibtion vector using the full-search 

block-matching motion estimator, to calculate? the motion yeqtorSi , This ^algor^ very 
cost effective approach to the motion estimation problem, but the more effective ones are 

10 based on the same principle. 

In block-matching motion estimation algorithms, a displacement vector is 
assigned to the center X = (Xx, Xy)*^ of a block of pixels B(X) in the current field n by 



1. 

8 



searching a similar block within a search area SA(X), also centered at X. but in the previous = 
15 field n-1. The similar block has a center, /which Js shifted with respect to X over the , <C 

displacement. vector To find D(X number of; candidate vectors C a^re 

evaluated applying an error measure (C, X, n) to quantify block similarity.* More^formally, 
CS"™* is defined as the set of candidates C, describing all possible (usually integer) 
displacements with respect to X within die search area SA(X) in the previous image^: 

20 

GS^^* = {C'l -N.:< Q:<:N, .M c< Cy < M . . r ; y , > <1) 

where '^I and M are constants limiting SA(Xy. Furthermore, a blobk B(X) centered at X and 
of size X by Y consisting of pixel positioiis jf =" (k.y)''' in the present field n, is no^ 
' 25 considered:" "- ' ' ' ' ' '■' ' "■' ■' - • 

' B(i) =' {'^ fXx -'x/2 <^ x"<^'x1 + X/2"'a x/-'y/2'^ y _< "Xy' + ' Y/2}" ' ' ' ' • (2) 
The displacement vector D(X, n) resulting from the block-matching 
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process, is a candidate vector C which yields the minimumivalue of aiif error function; 
(C, X. n):'"' 

."%;•'•-'>. r I.- • .• • ■ • -.'Mr'-' ' •/ , j -yr: j-.i:^^ 

B(k^ kfW '{C € CS'*** I '€(c';i; n) ^ €(P^.X. n) vt^ € eS""*}^ ' " ' " (3) 

r.- . Ifv Wjhiich i? ,*^^^^^ ^he smallest 



Vjc 6 B(X): D(x, n) = 5(X, n) ' ' ^ (4) 

10 



o 

yj rather than to the center pixel only, a large reduction of computations is achieved. 

^ As an implication, consecutive blocks B(X) are not overlapping. The 

<C error value for a given candidate vector is a function of the luminance values of the 

^ pixels in the current block and those of the shifted block from a previous field; sumrried over 

^ 15 the^^block B(X). A con choice, which we will us'e,^ is the sum ofHhe absolute^ ' - 

fltt . :;; i;'di£fierences: ri : ; , - ■■ • i- {}\ ,X /V- ... . ^^v. r'^.: .. - --- j 

€^ (C, X, n) = |F(jc, n-1) - Fix + C^, n)| (5) 



20 although a mean square error or a normalized cross correlation function are sometimes 
proposed. 

r . • , The difference between; the first and the second motion estimator, using 

candiidates Cb,and C, respectively , and resulting in motion vectors Db and Df, is that we 
propose in the method according to the invention, lies in the error criierion .of (5) . According 
25 to (5), the current field is shifted to optimally match the previous field (backward 

prediction). The second estimator that we propose is identical to the first, apart from (5) 
which is replaced by: 
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€^ (C X, n) = 53 \F{x,n) - Fix - " " * ' (6) 



This implies that Z)f is optimally suited for forward prediction.' With two 
estimators defined above, a high match error of the best matching candidate means that it is: 
5 • difficult to predict the current field with the previous, for the forward predicting 

estimator applying (6), or that it is: ' ^ ' 

• difficult to predict the previous fifeld with the present, for the backwards 

predicting estuhatbr applying ^^(^^^^^ * * 

10 The first situation occurs in case of uncovering background, the other is 

typical for a covering of the background. Th^ rdative perfbrmahce RP of the two motion 
estimators is now proposed to decide upon covering and uncovering: 



u 

a 



ijp=_L* fL- - (7) 



Where 5 is introduced to prevent decisions of tovering/uricovering in case of small match 
: : errors. Inrthe;experinients 5. was. set to approxirnately ,L percent of the maximum error. 

20 However, preferably the following decision is ta^^^ 

decision = ( ^ ^ * 1/77. 4.71. ^ ^^'^^^ 

where th, is inti-bdiiced to prevent decisibhs of coveirihg/uncbvermg' in dase bf small match 
errors. In the experiments Th| was set to approximateiiy^3 and ThV to'2b.' ' ' 

25 

' / When a. block is detected as belonging to either thexoyering or, the 
uncovering region, the 8 blocks surronding it are also assigned ttf that region, in order to 
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i 



improve the spatial consistency. ..... . 

Fig. 1 illustrates the covering and uncovering background situations. A 
moving object, a circle, is shown moving (with motion vector J5) from left to right. N 
.5 indicates ^he position in field n, while N + 1 indicates the position in field n+ 1 . Region I is 
uncovered in field n+1, while for region III covering occurs. Region II can be interpolated 
ft-om both fields. 

Although this brings some clarity about areas where covering and 
uncovering appears, the precision of the method is not optimally, sijjted for the image(s) to be 
10 interpolated. In that image only part of region I is uncovered and only part of region III 
becomes coyered. The fraction depends on the temporal position of the interpolated image 
- relative to the neighboring original images. If the interpolated image has to be valid at the 
temporal instance exactly in between the previous and next original, than the covefiiig and 
uncovering areas are half the size that we detected. 
15 S'i 

Fig. 2 illustrates this case into more detail, and shows, particularly, 
regions in which covering and uncovering occurs, and where in the interpolated image which 
motion vector can be safely used, and from which,, nexcor previous, image data should be 
used in the interpolation process. Region la can be:interp9lated.from,,the next -image usin]g 

20 vector D.. Region lb can be interpolated from the next and the previous image using vector 

' Db-iRegion . Ilia. can be intesrpolatedy from the next; and the, previ^^ 

Regioh Illb caii be' interpolated from tHe ftext' linage using vector D?. Two areas' still exist, 
i.e. la and Illb, for which no suitable motion vector has been identified. Hereinafter, a 
practical solution for all regions will be introduced, = * 

25 

So far, we distinguisHed^areks in which covering and uncovering occurs 
between two successive images. Howev6r,^ at ihe temporal instance where the new image is 
r interpolated we need to refine our earlier recognition of covering and uncovering based on €5 
and €f. To this end,, we define fb' and according to: . ^ ^ ^ 

30 



IN 



BI 



(9) 



BNSDOCID: <WO 992gS2QA2 I > 



wo 99/22520 PCT/IB98/01594 

8 

■and i''"- . ■■ '• I -J ;- .• • .:'■=" (.-. .■• .' 

e; n) '- ■[• £ -{iiXif^a^^ w-l)''---F(jr.-+^<l-a)5> n))tW^ % 'O-^v .x-, (10) 



' * ' an 



where Ngi stands for the number of samples in block B,(X). Instead of a fourth power, a O 
second power can also be used. The block B,(X) is not necessarily identical to BCX), A LU 

5 smaller block size >yill mostly be preferred, as this allows a more detailed region ^ 
categorization. Th^ error measure has been chosen since it has proven to be tfief mosit ^ 
effective, in most cases. ' ' ^ : a . * ? ^ 

While we ^e within region I,, it is conpluded, if e,^' <. Th, that we are in 
region lb, and in region la otherwise. The variable Th, here, is an experimentally 

10 determined, small, threshold value. In words:' we assume to be in region lb when 1)^ can be 
considered valid. This occurs, when, at Jhe f.einporal positipn of the image.^ to be Jnterpolated, 
the two luminance values obtained according to Dj,, i.e. the one from the delayed field and 
the one from the current field, can be matched with a small error. We assume to be in region 

la when cannot be considered valid. 

13^ ; \u. : ? J Similarly, while ^ye a^^ HI, :we cpnclude that if 6/ Th, 

that we are in region Hla, . and in region Illb otherwise. In.words: we assume tp be in region 

Ilia when Df can be considered valid. This bccuts when, ^ at the temporal position of the ^ 

mage to be interpolated, the luminance valiies bbtained according to Df can be matched with 

a small error. We assume to be in region Illb when Df cannot be considered valid. 

20 f - ^ ... ...^ ... , V 

It can be concluded that for any image to be interpolated between the 

previous and the. current field, it seems best to interpolate the result from the previous field 

.only, or, mainly, in case of 'real covering' of the background (region Illb), whereas in case 

of 'real uncovering' of the background (region la), the motion compensated data from the 

25 next field is preferred in the interpolation process: In all other cases; the motion compensated 

interpolator can use; the data^ from both fields. ^.^^ : 



So, where a pripr art motion compensated average would yield a 
intermediate picmre according to: 
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FjCJE; II) = 1/2 (F(jc - oD(jf, «), n-1) + F(if + (l-a)5(x, n). «)) >:-,^ (11) 

where a determines the temporal position of the interpolated image, we now propose to use 
different interpolation strategies for the various regions recognized according to the previous 
section. For example: 



I . F(jc;r a »)' « - 1) . , region lU^ 

F(jc + (1 - a)^,(jc. n), n) , region 



I 



CD 



i(F(jc- dDX^/n)/ n - l) + ' • v region III^ 

2\ . , . . (12) 

^|(>(jf- aD/jc, n); rt ^ 1) +F(jf +^(l-^*a)5|tf; h^^^^ otherwise a ' t : : . 



' ' " ■ Rather then switching between the various interpolation methods in (12), a 
soft switch coiild be thought of, i.e. a'fadihg between the five alternatives.. We will,.:.* 
10 • however, hot further elaborate .this thought here, - . , /. : ^ . / . . 

. .. ^ In (12), and al5p in Fig. 2, vectors D? occur,, which up to now have no 

meaning. They are required for those parts of the interpolated image where there is either 
covering or uncovering. Therefore it is known that either the previous or the next iniage 
should be solely used in the interpolation, but no valid vector is available to shift the 
,15 correspoiiding background that either appears or disappears. 

In order to solve this problem, we here propose to^'seairch'^ fbr the 'nearest 
vector describing the background velocity. If we are in the uncovering region I, this nearest 

...^ vector is. obtained. from. the backward predicting motion estimator, generating D^, at a 

location shifted over the negative value of ^he^forward predicting m^ vector -D^,. 

20 multiplied by a, the temporal position of the image to be interpolated. If we are in the 
covering region III, this nearest vector is obtained from the forward predicting motion 
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> 
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3 



estimator, generating D^, at a location shifted ^over the negative value of the backward 
predicting motion vector JD^, multiplied by (1— a). So, more formally: 

/ . . . . , V : . . . - • ^ 

• ••' . .r , . i.«,"!VVS^ : ■■i- ..... ^ . 

J D. (x - ixD^, n) , (regiori t y Liil 'real uncovering^ 
^ ' P^^^ + (1 ^ n) , {region III^^ i.e. 'real covering') 

5 ,-^U).Vr^. 

An alternative, prior art, motion compensated interpolation was described 
in [1]. It uses an order statistical filter to make the upconversion more robust with regard to 
I i i erroneous mbtioh vectors. A very basic Version wiais described in [6], and applied in the 

CD Natural Motion IC [9]. This version uses: 

IQ.. 

a !? t\iX^y :^) 'f: "}^^^^i^^^ "^)\ '^rP^^^ Wit: CV - aXPC?^ "X n)), . (14) 

''^^^ with . ~ ^c'^r^' :\ .i. : 'Z ^ ^ ' ..-a ' ' . ''-j.' • *' . ; 



and 

a , (b^a^c V c^a^b) 

15 me J (a, i>, c) = ^ 2? , (a<:*^c V c^b^a) (16) 

c , otherwise 



In regions lb and Ilia a safer version of the interpolation, according to the 
current invention would use a similar aoproach. As can be seen from the equations (18) 
below, motion compensated weighted ayer^ging is proposed for 'uncomplicated', areas. In 
20 region lb and Illa, i.e. the areas m which there is no acmal covering in the image to be 
interpolated, but yet near a complicated region, the robust interpolation according to (14) is 
proposed. 

In the 'rear covering and uncovering areas, la and Illb, we would like as 
well to propose a robust interpolation method. Thus we again make use of a median filter. 
25 As an input to the median filter, our best guess is the pixel from the previous/current frame 
(depending on covering/uncovering), shifted over Do. Another useful 'candidate pixel' is the 
pixel selected when shifting over the vector that would be valid for the neighboring a/b area 
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(la, in case we are in the uncovering region, Illb otherwise). The third pixel that we propose 
here is the one from the same field shifted over D-,?, where: 



D„(jC, 72) = 



(x - Df^, n) , (region i.e. 'real uncovering^ 

(x^ ^^ p^y n) . {region III ^ i.e. 'real covering') 



(17) 



> 

o 

lil 



CO 
CD 



or preferably. 



D. (x - ia+ — a)£>j,, «) , (regioh I^, i.e. 'real uncovering) 

J -r J'. ; i-.S^,, v. -^.-.f rr: ..■ •<■;..•••■. .■■ ■ . 

2 

(jc- + ((17a) +—a)Z?y„n),-,^ (region 7/7^, 'rca/ coveri/igO 



■ (17a) 



The thought behind this choice is that D, = D« in most cases where 0/is 
'■ • reliable. In those cases therefore our best ^guess. is actually selected as'the dutpiit pixel. If D7 
10 and are not identical, it is likely that confusion exists between region a and b. and the 
chance increases that the second 'candidate pixel' is chosen. Eventually the sequence would 
be processed according to: . 



15 



FXx, n) = 



med {F{x - aDM n), n-l), Fjj,^, F(X - aD„(Jf, n), , region III^ 

med {Fix + (1 -a)D,(jc, n), n)» F,^ F(jf + (l-a)D,,(je'^ «), «)) , region 

med (Fix - dD/xJn), n-l),'Avl F(X + '(i-lx)D/x, ri), h)) , region III^ 
■ 'med (F(,t -^aDfjix, n), n-l), Av, FOc *'(l-a)D^ix, h), ^ 7/ - • 

■ - (Fix - dD/x, n), n-l) + F{i ^\\-a)D^{X, n), n)) otherwise" " 



0 r 



where:rj ^: V.> ; ; 



F^' = '- (F (^ - a^ijf, it),'h-l5 '+ U '^''(\-cL)Djik,^n),n)) 



(18) 



(19) 
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and '■ " '■" • 

In order to provide a safer interpolation strategy for the most critical regions (la and Illb) an 
5 other solution has been tested. 



{8 



For jegion Ij,: 



5 n) = med^'(Fix +,(1.7 a)^7(J^ .. k v. . • 

3 F(jc + (1 - a)5„(J^ n), n), 

^ Av, (18a) 



10 



and for region Hlb: • > 



F(jc + (1 - a)D/x, nx 'n)) 



This sophistication experimentally proved to significantly increase the 
interpolation quality. A preferred embodiment of the current invention is shown in the block 

15 diagram of Fig. 3. An input video signal I is applied to a dual estimator DE Having two 
motion vector estimators MEl,' ME2. In the dual estimator DE, the input video signal I is 
applied to a field delay FMl, a first input of the first motion vector estimator MEl','^' and to a 
first input of the'second 'motion vector ^^e ME2 thfii a shifter S2. An oUtput signal of 

the field delay FMl is applied to a second input of the ;SeGpnd mot^ and to 

20 a second input of the first motion. vector estimator MEl thru a shifter, SI. The first motion 
vector estimator MEl supplies the vector Df, while the second motion vector estimator ME2 
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supplies the vector Db. 

The input video signal I is also applied to a motion-compensated 
interpolator MCI. In the riiotiori-compensated^i MCI, the'input video signal I is 

applied to a field delay FM2, and to a shifter S3, An output of the field delay FM2 is 
5 applied to a shifter S4. The shifters S3, S4 are controlled by the motion vectors Db, Df 
received from the dual estimator DE. Outputs of the shifters S3, S4 are applied to a median 
circuit med and to an average circuit Av. Outputs of the median circuit med and the average 
circuit Av are applied to a multiplexer MUX which supplies the output signal O to a display 
device CRT for displaying the output signal at, for example, a 100 Hz field fate ! The motion 
9k 10 vectors Db, Df, their errors eb, 6f, and threshold values ml, m2 are applied to a region 

O detector RD which furnishes a control signaMd th^ multiplexer MUX; - '^^ - / 

yj A-^^ /v ^ ' -^'J 



CD 



A primary aspect of the invention can .be sununarized as follows. In 
pictures resulting from motion compensated field r?ite convertors an -artifact is visible at the 
15 boundary of moving objects where either covering or uncovering of backgrounds occurs. It is 
the object of the present invention to eliminate this local artifact. To this end, areas where 
covering and uncovering occur are recognized based on the relative prediction quality of two 
motion vectors. The two motion vectors are estimated between the same field pair, )but differ 
in that the first is calculated to optimally jpredict .the previous fie]d. from the next, whereas 
20 the second tries to predict the next field from the previoiis. An upcpnversion method using 
the two motion vectors is also described in this disclosure. 

In a preferred interpolation method, two rnotioh vectors, one predicting 
the next image from the previous image and the other predicting the previous image from the 
next, are taken fi-om both sides of a discontinuity in the vector field, and both applied to 
25., . predict, the next image from the previous and to predict the previous image from the next, 
..and, a choice between the vectors^ is rnade in dependence on the relative suitability for the 

. , teste4,prediction.^.,. _ .../.n.^,- t/^^^,- >• : v... 

A preferred embodiment can be summarized as follows. An irhage 
^ ; between ^neighboring images in a sequence,^ is interpolated by applying two motion vectors 

30! y K^T^and Vy, one; optin\ally suitedkfor backwards. predicting and the.pther ior fory/fj^s 

prekiiiiioMhe^dribo^ Ivux^ij . a . \ ' t 

• means to shift data from the next image over a fraction of , 
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• means to shift data from the previous image over a complementary fraction of V^, 

• means to shift data from the previous image oven a fraction of V^, :■ . . . ^ 

• means to shift data from the next image over a fraction of K^, ' * jj - 

• , means to combine the shifted daia,; and possiJ>ly data in dependence on 
segmentation information indicating covering/uncovering in.the intepjolated jmage, in order 
to form the interpolated.image^ characterized 4n that in case of covering/uncovering,. at least 
one of the vectors that are used in the combination process ,aa;e, taken from.a.s^^^ position 
,that 4iffers -from^A^ pp^itipn.fc^r.y^^ calculated. Preferably, the position 
from which the vector is taken, is shifted, in. jhe opposite direc^^^ in which the vector is 

pointing, with regard to the.po$itipn for,>yhich it wa? calculated. Advantageously, in case of 

■ " ■ -■ ,^ , „. } •• . ■ 

covering at leiast aV^ is used shifted, oyer »^ajf^;i:while in case of uncovering at, least aV^ is 
used shifted over -aJ^. 

. . It should be noted diat the above-mentioned enibodiments illustrate rather 
than limit the invention, and .that those skilled in the art will be able to design maijy 
alternative embodiments without departing from the scope of the appended claims. In the 
claims, any reference, signs placed bet\yeen parentheses shall, not be construed as limiting the 
claim. The word "comprising", does not exclude the^presence of other elements or steps than 
those listed , in a claim. The invention can be implemented by means of hardware comprising 
seyeraL distinct elements, and by means pf a suitably programmed computer. In a device 
claim reciting a plurality of different means, several of these mMns may be embodied by a 
one and the same item of hardware. In an .alternative embodiment, the covered and 
\unco verged .pai:Ts are not detected by comp2^"ing b?ickward^ and f estimatipii errors, but 

by analyzing direction and magnimde of motion vectors around contours, ^ ; 
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1. A method of detecting covered and uncovered 'jiam in a (il) to be 

' - interpolated* betvveen neighboring previous input' images, the method 

••"^xairiiprisiri^^^ /-io* v^'- achoni tf^^/i^^^ .-a; 'Jh ry-j^-^a. i:;ay! 

detecting motion from said next input image (IH) to said previous input 
5 image (I), resulting in. backward motion vectors. (D^) having _ backward 

estimation errors (€b); ■ ^ r ■ '^^-^ - • 

bT. .v -a ^ :„r;rd^^^f^^i^g n^o^io^ input image (I) to said next input 

.image (III),, resulting in= forward motion fm:^toj^^(DJ) Jliaving| corresponding forward . . 
■:f^r./7:i?stima^ >.v', nr-;r v . ' . • ■. '^-^v/.-, 

IP : i. detecting uncovered (I) and covered said neighboring . . 

previous (I) , and nex^ 
;^ : ra . ? \ * : ... said method being characterized by the further steps of: 

. : - detecting in said thus^^^..d^^ (la) in 

; ; r said image; (II) - to he interpolated ^)y. det^^^ backward .estimation eirors by 

15 comparing bt)|tb neighboring previous^ (^^^ input images when partially shifted 

oyer said; backward motion yector^^ (Pjj):^Q a, teniporal .location of said image (Il).tOj be 
interpolated, and by comparing said second backward estimation errors to a threshold; and 
: detecting in said -thus detectpd covered parts (III), covered parts (Illb) in 
r * said image (II) to. be inte^50^ted by ^etennining-second forward estimation errors by 
20 comparing both neighboring previous (I) and next (III)^ input images when partially shifted 
: oyer , said; fprwa^^ location of said image (II) to be 

iflteipolate,d, ;a^^ errors to said threshold. 



2. A method as claimed in claim 1, wherein said step of detecting said 

25-' yncovere^^^^^ (1) andrsaid covered (III) parts in said .neighboring previous (I) and next (III) input 
image§;includes the step of processmg •said.^ba^^ forward 
estimation errors (6f) to detect said uncovered (I) and said covered (III) parts, in •said. 
. ; neighboring previous (J) and; next (III) input: im^gps>vrespe.a^^^ 
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3. A device for detecting covered and uncovered pans in an image. (II) to be 
interpolated between neighboring previous (I) and next (III) input images, the device 
comprising: 

means for detecting motion from said next input image (III) to said 
5 previous input image (I), resulting in backward motion vectors (D5) having corresponding 
backward^ esiinia^ 

vf: ^:;? i . st^^- (.means fofi,deteQting:ni^ preyiouSiinpw.Amage/(I)-ito said next, 

input image (III), resulting in forward motion vectors (Df) having coriiespondingt fpi^ 

:^/^':-iiestimation errors : v( >k zv^n ^ite ?r-"'Vr 

10 ' • ' '^'^ ' ' means for detecting uiicoverkl (I) and covered (III) parts in said* - 
^ neighboring previous (I) and next (III) input images, respectively; ^ ^ ' . ■ ' v 3 

;\ , ; i;; meaus for detecting ih's^^^ parts (I), uncovered 

O parte' (la) in S2ud image (II) to '6eHhteip6lateci by deterinihing second backwairci estimation 

UJ errors by comparing both neighboring previous (I) and next (III) input images -wheil panially 

15 shifted over said backward motibii vector^ (D J tb a temporal location of said image (II) tb be 
interpolated, and by comparing siid second backwatd estimation errors to a thlt'eshoM; 

and detecting in said thus detected covered parts (IH), covered parts (Illb) in 
-said image (II) to be interpolated by dHeiffihiiiig second 'fofwartf estimation errors by 
' bbmparih'g both neigHbbririg prevrdus^tl)^1afid"hext (^^ input' images \vhen partially shifted 
20 over *'said' forward motion vectors (Df) to a' tem^^ location -of -said image (II) to be 
interjpblated, and by comparing said secbhd forward eitiihatiori' errors W said threshold. 

4. " ' - A device for estiihatirig niotioh v^ 
' a device for detectirig ^covered and uncovered pSarts in an image to be 

25^ interpolated as claimed in claimi^; and- ^ ^ ^ * ' ' r^-^ -r-^u. ^ ^r.^-.n ^ ^ t 

' " ' ' ' * ' * ^^^means for furnishing mbtibn vectors for said covered' and^tocovered parts 
' ih'Mid image (II) td be interpolated in dependence upon said- back w (D^) 
and said forward motion vectors (Df) . 

^30^ 5.' ' ' '^'A 'methoa^^o^fresUm^^^ iribtibn vdctdrs' for 4n image <II) to 
' bei^efen^neighbbring-pr^^^^ niethbd cdrhprising the 

dl^fefcting "irfoti'dh ndkt input image (III) to said previous input 
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ima[ge :(I)v resulting-: in ^^b^^^ moiiori.vect6rs (Db) having corresponding backward 
estimation errors (€b); '-i^^ . -V'--'- ' 

detecting motion from said previous input image (I) to said next input 
- "-image (III); resulting' in fd having corresponding forward . 

5 estimation errors (€f);> • ' ^hrjixiC'.\. i^^ii^^^i: n / .'^-r:.;...:. i ;:v:i.. ; r, 

r:I >tj;£>i ^ i:: . . .? detecting uncovered (la) and covered: (IIIb):pans in said image (II) to be 
interpolated; 'r.i.i-'.\ n^,, v:A:5ii= 

S^^;d vfumshihg^ittOtiott vectic^ (la) in dependence 

^ upioii ?said forward mdtion - vectors • (Df) taJcetf. frdtfi' Ibcatibiis iwhoUy ahd/dr partly^ shifted over 

^ 10 said backward jnotion; vector:(Di); a^ r^^i^^yv Kr^^'^ ^^;:w; ■ {■■ ' .r;^; : - : 
O furnishing motion vectors for said covered pans (Illb) in dependence upon 

iLj' .ii: r said backwiard motion vectors (D|^) tskeWfrbih'JdCatidh^ whdliy dnd/or partly shifted over 
siaid^ 'forward nlotion vectors-(D,)l-:v''^ -J-i) ^>^^^ out (!) v.;:;!:^ u^:^v.^^^:; 

b-'^j.' ■; ■ : ' r- y ^rv-; .Ct ':c;.rr{ gar. ? . •> 

15 6"; • ^ ' ^ A method as claimed in 'cMm Sv Avhe^ of detecting said ' 

uncovered (la) and said covered (Illb) parts in said image (II) to be interpolated includes the 
step of processing said backw said forward estimation errors (6f) 

to^etect said uncovered (laO arictsaitf tdvefed^ (Illb) parts in said iniage (II) to be / 
interpolated. ; > . : t si; J -7 

7. A method as claimed in claim 5, wherein said steps df fufnishirig motion 

Vectors for said uncovered (la) M covered (lllb) parts compirise the steps of: 
i::// rii ; . \. furnishing first mbtidnvett^ said uncovered Mparts (la^by taking said 

forward motion vectors (Df) from locatidns wholly shifted over said backward motion vectors 

; I V c; :i /.v ; funiishing sccohd mdtioh vectdrs -for said uncovered parts (ta)'by taking 
said forward motion vectors (Df) ftbinidc&tiohs partly shifted ^^b^ 
vectors (Db) corresponding to a temporal location of said image (II) to be interpolated 
between said neighboring previous (I) Snd next- (Ill)iirnages; • ' 

30 iS. ; : r' / fiirnishing fi^st motion vectbrs'^f or 's^^^^ parts (Illb) by taking said 

backward rridtibri vectors' (Db)^ from Ibcatioiis wholly- shifted over-said forward ■^mdtion vectors 
(Df); 

^ ^- ' - - ftirniishing secdndWotioh" vecTob^ parts (Illb) by taking 

•said 'backward motion vectors (Db) ■frbm^ locations jpirtly- stofted said forward^ motion 
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veetors (Df) corresponding-, to a. temporahloGationvOft said iniage.(II«) to;be:.^imerpolateci 
between said neighboring previous (I) and next (III) images, >^ . i ^ c rx 

8'. ru: ; . ^ o - . k^.:A of: interp.oIi2^ting an/image : (II) -between:neighbpt^^^ (I) 

5 and next (III) input images, the method comprising the steps of: - : n>r , • . i 
, /Tsr.:^ ; I'i ' .v,estinM]dngv:mptiQniYqptots for, sai(^.iimage: (II) to be interpolated in 
accordance with claim 5; and s o rv vi: 

v\:.ttn:p:j (c^) H-j5ifornishingysai4:image#^^ t^^injerpjplated by; means of said backward 
; ^ ^ m6tion7y€?ctor&/.(Db)i 5^id :f<)rward^mptiOm5^e^torsi(D^^^^ fOr -saidj:, 

10 uncovered parts (la), and said motion vectors for said [Covered parts (Illb). :; '.i,, /^^ ; u-- p , 

■.o9 ./ 1 . ^: ;v:.:A device for estimating ^nq^ioniVQCtors an image (II) tQ;-berinterpolated 
between neighboring previous (I) and next (III) input images; .the device comprising:: 
means for detecting motion from said next input image (III) to said 
ULI 15 previous input image (I) ^ resulting in ba:cfcward mo^^^ (Pb) having corresponding 

■ . backward. estimation errors (€b); . . ■' .r ,\ ^ . ; - ? . . < . 

—J:. ) ] \ , - means for detecting motion from :said previous image (I) to said next 

S>» image (III); resulting in forward motion vectors (Df) having, corresponding forward, ;: 

|- estimation errors (€f) ; : . : - 

Jjj 20 means for detecting uncovered (la) and covered (Illb) parts in said image 

^ ; :i (II).t6 be interpolated; ^ r / ^ : : o v ' 

J " ; means for furnishmg motion vector 
. .V : ' dependence upon said forward motion, vectors (Df) taken from, locations wholly and/or partly 
v shifted oyer said backward motion vector, (Db); Mid , v .v. - . "jc r . <^ 

25 means for furnishing motion vectors for said covered parts (Illb) in ; 

: J - dependence^^uponssaid backward niotipn vectors XPji) taken from locations wholly and/or 
>-'partly shifted^Qyer said forward motion vector /(Df). / 



10. A display apparatusv^comprising: ^^ - r ; • , / . .. ^ r^^^ ; 

SQr,; - .1. (ou } ' deyicC; (PE),;fOT yeciors^,for an image (II) to be , . 

. : ^ . ;!,inteipQjated between^^^n^ (III); inputs images^as defined in claim 

9; 

(dllh y^tc^ r^iri rneans;(MC^):fi^ image (II) between said neighboring 

:,J:o preyious.XI);and nextv(HI) inpm of said backward motion vector%(Db)» said 



BNSDOCID: <WO_9922520A2J_> 



wo 99/22520 



PCT/IB98/01594 



20 



forward motion vectors (Df), said rnotion vectors for said uncovered parts, and said motion 
vectors for said covered parts; and 

means (CRT) for displaying an image sequence formed by said 
neighboring previous (I) and next (III) input images and said image (II). 
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(54) Title: MOTION VECTOR ESTIMATION AND DETECnON OF COVERED/UNCOVERED IMAGE PARTS 

(57) Abstract . \ 

: / ■■' [ 
In a method of detecting 
covered and uncovered parts in an 
image to be interpolated between 
neighboring previous (I) and 
next (III) input images, backward 
motion vectors from the next 
input image to the previous input 
image, and having corresponding 
backward estimation errors, and i 
forward motion vectors from 
the previous input image to the . 
next input image, and having : 
corresponding forward estimation . 
errors, are determined, uncovered 

and covered parts are detected * ^ , ^ i ^ * » n \jr\ 

in the neighboring previous and r Qjj ™J^...u-.......j^^^ l , » | ^ 

next input images, respectively, in ^ ' , , . ' ' « » s i \ \ \ 

the thus detected uncovered parts, . " 
uncovered parts in the image to- ' ? 

be interpolated are detected by \ 

determining second backward estimation errors by comparing both neighboring previous and next input images when partially shifted over 
the backward motion vectors to a temporal location of said image to be interpolated, and by comparing the second backward estimation 
errors to a threshold, and in the thus detected'covercd parts, covered parts in the image to be interpolated are detected by determining 
second forward estimation errors by cbmpanhg both neighboring'previous'atid hefxt input "images when partially shifted over the forward 
motion vectors to a temporal location of thie;,.ii]nage..,tQ ibe interpolated, and by compariiig the second forward estimation errors to the 
threshold. ■ ? 
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